Content licensing organisation NLA Media Access has launched a free service to help publishers identify organisations scraping their content and pursue either commercial licensing deals or legal action.
The Collective Intelligence Advisory Service analyses website data from participating publishers to identify activity that may indicate content is being scraped without permission.
Publishers may be able to generate “significant” revenue in the long term, said NLA chief executive Henry Jones, through licensing agreements with organisations that are identified as using their content. They could also seek legal action with NLA support.
Read more: Publishers to bill AI firms for unwanted scraping – and take them to court if they don’t pay
The launch comes amid backlash to Meta’s planned terms and conditions update which would effectively give it sweeping rights to scrape content from any publisher that has an account on Facebook, Instagram or Whatsapp.
Jones noted that after multiple international publishing bodies joined together in opposition, Meta “immediately climbed down and delayed” its plan. It has now pledged to undertake a full consultation before making any changes.
He said: “This is a similar thing where if you’re stealing content from one publisher, you might get away with it, but if you’re doing it on 14,000 titles, then we’re well placed to go after them.”
He added that anyone stealing and monetising content without publisher permission “must legitimise their activity or face the consequences, and NLA is well positioned to manage both outcomes”.
“We will advise publishers on the trends we discover from aggregating their data, and help them capture the value generated by their journalism through commercial agreements or legal action.”
How does it work?
The Collective Intelligence Advisory Service builds on NLA’s existing Text Tracker, which helps publishers spot copyright infringements of their content and has taken down more than 150,000 pieces of infringing content in the year to date, including more than 40 full site takedowns.
Jones said the service was designed to identify scrapers that can be difficult to spot from a single publisher’s website data.
“It’s essentially looking at who is accessing your website, and a lot of those will be readers like you and I,” said Jones. “But some of them will be wholesale scrapers of content, or crawlers or bots that are not adhering to particular publishers’ robots.txt agreements.” He added that these can be disguised as regular readers.
‘Robots.txt’, the Robots Exclusion Protocol, is a defence mechanism against crawlers designed to stop automated software from crawling sites (as opposed to human visitors). It is not legally binding and is more akin to a ‘gentleman’s agreement’, which is routinely ignored.
NLA ran a pilot with UK regional publishing giant Newsquest and three nationals, including The Telegraph, to test whether it could spot illegitimate users more quickly.
“Let’s say we’ve got a spoofed iPhone access log which has read 400 articles on the Telegraph website, that could, for all intents and purposes, just be a very avid Telegraph reader,” Jones added.
“But if we see them reading 300 on one of the Newsquest sites and another 200 on the Guardian, it almost certainly is a bot,” said Jones.
AI scraping: To licensing or legal action
NLA already sells licences to organisations allowing them to scrape publisher websites for a fee, but there are restrictions on how the content is used.
“We sell a licence at the moment that allows organisations to scrape publisher websites for a fee, and there’s very narrow guardrails on exactly what they can do with that content,” said Jones. “If they resist and don’t stop doing what they’re doing, then we will have all the evidence we need in order to pursue some sort of legal action.”
He added that the pilot found examples of companies in Eastern Europe and Asia-Pacific where “they just thought they could take the content”.
“When we’ve gone to them and said, ‘you can’t’, they go, ‘okay, well, we’re going to need to buy a licence then, send us the terms’.”
NLA represents 14,000 titles in the UK and believes collective action could make legal proceedings more affordable than publishers pursuing legal action individually.
Jones said he expected unauthorised scraping to be resolved through pre-action legal letters, but “in the next 18 months we will find examples of organisations that want to resist, and then we will end up going through with a formal collective action”.
Email pged@pressgazette.co.uk to point out mistakes, provide story tips or send in a letter for publication on our "Letters Page" blog
